Processing scene objects

ABSTRACT

Apparatus for generating image data, including a comprising memory means configured to store data defining a volume having comprising a plurality of objects and at least one viewport, a memory means configured to store motion data for each of said the objects, configuration data for said the viewport and instructions, and a processor processing means configured by said the instructions to perform the following steps of, for. For each of the objects a position is calculated along its motion path at an interval of a user-specified time period, wherein the interval is dependent upon the shutter length for the object; the . The object is then translated to this position; the. The objects are then rendered through the viewport to produce a sample; and when. When a specified plurality of samples have been rendered they are blended to generate image data wherein each object is independently motion blurred.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to generating image data as theblending of a plurality of samples.

[0003] 2. Description of the Related Art

[0004] Advances in the field of digital signal processing have allowedmany new developments to be effected in video and film post production.Many sophisticated image modifications and special effects have becomewidely accepted as part of post-production procedures.

[0005] Often, post-production techniques are used to generate additionalimage data to be composited within frames of original video or filmfootage, either because said additional image data cannot berealistically committed to movie or video film, for instance if themovie script requires an actor to jump over a mile-wide gap, or becauseit does not exist, for instance if the movie script requiresphoto-realistic alien spaceships.

[0006] The degree of realism conveyed by such image data istraditionally paramount in order to immerse an audience within thenarrative, and techniques are known with which to motion-blur theadditional image data when such data portrays moving objects or actors.Motion is usually a function of the position of an object changing ineach frame of a sequence of frame. Displaying such a sequence at theappropriate speed, for instance 24 frames per second for movie films,provides the audience with the illusion of movement. Motion-blurringtechniques are used to enhance this illusion, and are especiallyrelevant when the object in question is moving at a fast pace. Motionblurring traditionally involves specifying a shutter length indicativeof the number of frames to use to evaluate the motion of an objectthereon, and a number of samples which defines how many snapshots aretaken of each of said frames, whereby said samples are subsequentlyblended and the output of said blending operation is an output frameshowing said object with motion-blur.

[0007] A problem however exists in motion-blurring techniques accordingto the known prior art, in that additional objects as described abovehave to be independently motion-blurred if artistic considerations orthe movie script requires discrete degrees of motion-blurring for eachadditional object in the same frame, for instance if two alienspaceships are flying at different speeds, because said motion-blurringtechniques according to the known prior art require the shutter lengthand the number of samples be configured for the entire scene or outputframe. This may generate visible artefacts which are highly undesirableto convey the required degree of realism.

[0008] Moreover, the independent motion-blurring of said additionalobjects is very resource-intensive, because a separate output frame maybe required for each of said motion-blurred objects, to be composited ata later stage with the original movie or video footage.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0009]FIG. 1 shows an image processing system operated by an imageeditor;

[0010]FIG. 2 details the hardware components of the image processingsystem of FIG. 1 in further detail;

[0011]FIG. 3 shows motion blur processing to generate image framesaccording to the known prior art;

[0012]FIG. 4 details the operational steps according to which a useroperates the image processing system of FIGS. 1 and 2;

[0013]FIG. 5 shows the contents of the main memory shown in FIG. 2 afterthe step of loading instructions and data shown in FIG. 4;

[0014]FIG. 6 provides an example of a scene structure including scenedata as shown in FIG. 5;

[0015]FIG. 7 further details the image data shown in FIGS. 5 and 6 as aplurality of objects, including a viewport, within a three-dimensionalvolume;

[0016]FIG. 8 illustrates the graphical user interface of the applicationshown in FIGS. 4 and 5, including a graphical representation of thescene structure and scene data shown in FIGS. 6 and 7;

[0017]FIG. 9 details the processing steps involved for editing scenedata shown in FIGS. 4 to 7 as scene objects shown in FIG. 8;

[0018]FIG. 10 graphically illustrates the step of equipping an object inthe scene of FIGS. 6 to 8 with a motion path as shown in FIG. 9;

[0019]FIG. 10A graphically illustrates a prior art clip of frames;

[0020]FIG. 11 details the processing steps according to which image datais rendered as shown in FIG. 4;

[0021]FIG. 12 further details the processing steps according to whichsamples are processed to generate motion blur shown in FIG. 11;

[0022]FIG. 13 shows the objects shown in FIG. 10 sampled and stacked inthe memory as shown in FIG. 12;

[0023]FIG. 14 details the processing steps according to which thesamples shown in FIG. 13 are processed to generate output frame imagedata;

[0024]FIG. 15 graphically illustrates a clip of frames produced by thepresent embodiment of the invention.

[0025] An embodiment of the invention will now be described by way ofexample only with reference to the abovementioned drawings.

WRITTEN DESCRIPTION OF THE BEST MODE FOR CARRYING OUT THE INVENTION

[0026]FIG. 1

[0027] Apparatus for generating image data comprises, in this example, apost-production station illustrated in FIG. 1. An image editor 101controls an image processing environment formed by a processing system102, a video monitor 103 and a RAID 104, by means of a keyboard 105, anda stylus-operated graphics tablet or a mouse 106. The processing system102, such as an Octane™ roduced by Silicon Graphics Inc., supplies imagesignals to the video display unit 103. Moving image data is stored onmemory provided by the redundant array of inexpensive discs (RAID) 104.The RAID is configured in such a way as to store a large volume of data,and to supply this data at a high bandwidth, when required, to theprocessing system 102. The processing system shown in FIG. 1 is optimalfor the purpose of processing image and other high bandwidth data. Insuch a system, the instructions for controlling the processing systemare complex. The invention relates to any computer system whereprocessing instructions are of significant complexity.

[0028] Instructions controlling the processing system 102 may beinstalled from a physical medium such as a CD-ROM or DVD-ROM 107, orover a network 108 from a network server 109, including the Internet 110accessed therefrom. These instructions enable the processing system 102to interpret user commands from the keyboard 105 and the mouse orgraphics tablet 106, such that image data, and other data, may beviewed, edited and processed.

[0029]FIG. 2

[0030] The processing system 102 shown in FIG. 1 is detailed in FIG. 2.The processing system comprises two central processing units 201 and 202operating in parallel. Each of these processors is a MIPS R10000manufactured by MIPS Technologies Incorporated, of Mountain View, Calif.Each of these processors 201 and 202 has a dedicated secondary cachememory 203 and 204 that facilitate per-CPU storage of frequently usedinstructions and data. Each CPU 201 and 202 further includes separateprimary instruction and data cache memory circuits on the same chip,thereby facilitating a further level of processing improvement. A memorycontroller 205 provides a common connection between the processors 201and 202 and a main memory 206. The main memory 206 comprises twogigabytes of dynamic RAM.

[0031] The memory controller 205 further facilitates connectivitybetween the aforementioned components of the processing system 102 and ahigh bandwidth non-blocking crossbar switch 207. The switch makes itpossible to provide a direct high capacity connection between any ofseveral attached circuits. These include a graphics card 208. Thegraphics card 208 generally receives instructions from the processors201 and 202 to perform various types of graphical image renderingprocesses, resulting in images, clips and scenes being rendered in realtime on the monitor 103. A high bandwidth SCSI bridge 209 provides aninterface to the RAID 104, and also, optionally, to a digital tapedevice, for use as backup.

[0032] A second SCSI bridge 210 facilitates connection between thecrossbar switch 207 and a DVD/CD-ROM drive 211. The DVD drive provides aconvenient way of receiving large quantities of instructions and data,and is typically used to install instructions for the processing system102 onto a hard disk drive 212. Once installed, instructions located onthe hard disk drive 212 may be fetched into main memory 206 and thenexecuted by the processors 201 and 202. An input output (I/O) bridge 213provides an interface for the mouse or graphics tablet 106 and thekeyboard 105, through which the user is able to provide instructions tothe processing system 102.

[0033]FIG. 3

[0034] Techniques are known to generate motion blur for an object ortalent to be composited in a clip of frames at a later stage, in orderto convincingly portray the illusion of movement at speed of said objector talent in the final composited clip. Generating motion blur for animaginary object according to the known priorart is shown in FIG. 3.

[0035] In order to generate a clip portraying, for example, a spaceship301 travelling at speed, said spaceship 301 is first modelled in athree-dimensional volume 302 with vertices, which eventually definepolygons to which textures and a variety of other characteristics may beapplied. The volume, or scene 302, is preferably equipped with aviewport 303, the purpose of which is to define a view frustum, theorigin of which functions as a camera to render a two-dimensional imageof the three-dimensional volume 302 and object 301 therein, as seenthrough the imaginary camera.

[0036] In order to generate the aforementioned motion blur for spaceship301 according to the known prior art, it is necessary to render aplurality of image frames 304, 305, wherein either spaceship 301 ismanually translated within volume 302 after image frame 304 is renderedsuch that image frame 305 portrays spaceship 301 at a differentlocation, or the position and/or orientation of viewport 303 is alteredafter image frame 304 is rendered so as to similarly obtain an imageframe 305 within which spaceship 301 has moved.

[0037] Upon obtaining the two distinct image frames 304, 305, acomposite image frame 306 portraying spaceship 301 with motion blur 307is obtained by sampling each of said frames 304, 305 a number of times.The number of frames sampled, which in this example is two, is known tothose skilled in the art as the shutter length, or the sample windowsize. In the example according to the prior art described above, fivesamples 308 to 312 are taken with a shutter length of two frames 304,305.

[0038] The samples 308 to 312 are subsequently blended, whereby the twofirst samples 308, 309 of image frame 304 carry less weight in thefive-sample average than the three samples 310 to 312 of image frame305, such that the latter position of spaceship 301 within image frame305 is better defined in composited image frame 306 than the previousposition of said spaceship 301 within image frame 304, shown as a blur307.

[0039]FIG. 4

[0040] The processing steps according to which the image processingsystem 102 of FIG. 1 generates image data according to the embodimentdescribed herein are further detailed in FIG. 4.

[0041] At step 401, the image processing system 102 is switched on. Atstep 402, instructions and the data that said instructions configureCPUs 201, 202 to process are loaded from hard disk drive 212, DVD-ROM107, network server 109 or the internet 110, such that said CPUs 201,202 may start processing said instructions and data at step 403.

[0042] At step 404, a scene is selected which comprises a structuredefined as a hierarchy of data processing nodes and a plurality of typesof data to be processed therewith.

[0043] The processing of said scene data according to said scenestructure generates at least one object within a scene orthree-dimensional volume configured with at least one viewport, wherebythe results of the editing of any of the data defining said object,scene or viewport may be rendered as a frame or a clip of frames at step406.

[0044] At step 407, a question is asked as to whether the scene data ofanother scene should be edited for subsequent rendering. If the questionasked at step 407 is answered in the affirmative, control is returned tostep 404, whereby the editor 101 may select a different scene structure.

[0045] Alternatively, the scene data rendered at step 406 as edited atstep 405 is acceptable for the intended purpose of editor 101, wherebythe processing of the instructions started at step 403 may now bestopped at step 408 and, eventually, the image processing system 102switched on at step 401 may eventually be switched off at step 409.

[0046]FIG. 5

[0047] The contents of the main memory 206 subsequent to theinstructions and data loading of step 402 are further detailed in FIG.5.

[0048] An operating system is shown at 501 which comprises a reduced setof instructions for CPUs 201 and 202, the purpose of which is to provideimage processing system 102 with basic functionality. Examples of basicfunctions include access to and management of files stored on hard diskdrive 212, or DVD/CD-ROM drive 211, network connectivity with RAID 104,server 109 and the internet 110, interpretation and processing of theinput from keyboard 105 and graphic tablet or mouse 106 and graphicaldata or binary data output. In the example, the operating system isIRIX™ provided by Silicone Graphics Inc, but it will be apparent tothose skilled in the art that the instructions of this embodiment may beeasily adapted to function with different other known operating systems,such as Windows™ provided by the Microsoft Corporation of Redmond,California or LINUX which is freely distributed.

[0049] An application is shown at 502 which comprises the instructionsloaded at step 402 and which enables the image processing system 102 toperform processing steps 404 to 407 within a specific graphical userinterface displayed on VDU 103. A scene structure is shown at 503 andscene data is shown at 504, which comprises various sets of userinput-dependent data and user input-independent data according to whichthe application shown at 502 generates image data.

[0050] Scene structure 503 comprises a plurality of node types 505, eachof which provides a specific functionality in the overall task ofrendering a scene according to step 406. Said node types 505 arestructured according to a hierarchy 506, which may preferably but notnecessarily take the form of a database, the purpose of which is toreference the order in which various node types 505 process scene data504. Scene structure 503 also includes at least one motion blurconfiguration file 507, the purpose and functionality of which will befurther described hereinafter.

[0051] A number of examples of scene data 504 are provided forillustrative purposes only and it will be readily apparent to thoseskilled in the art that the subset described is here limited only forthe purpose of clarity. Said scene data 504 may include image frames 508acquired from framestore 104, audio files 509 such as musical score orvoice acting for the scene structure selected at step 404. Said scenedata 504 may also include pre-designed three-dimensional models 510,such as a spaceship, and a variety of textures 511 to apply to saidmodels 510. In the example, scene data 504 includes lightmaps 512, thepurpose of which is to reduce the computational overhead of CPUs 201,202 when rendering the scene with artificial light sources. Scene data504 may finally include three-dimensional location references 513, thepurpose of which is to reference the position of the scene objectsedited at step 405 within the three-dimensional volume of the scene.

[0052]FIG. 6

[0053] A simplified example of a process tree, is shown in FIG. 6 as thescene structure 503 and scene data 504 loaded into memory 206 at step402.

[0054] Process trees generally consist of sequentially-linked processingnodes, each of which specifies a particular processing task required inorder to eventually achieve an output 601, under the form of acomposited frame or a sequence of composited frames. Traditionally anoutput scene 601 will comprise both image data and audio data.Accordingly, the composited scene will thus require the output from animage-keying node 602 and the output of a sound mixing node 603. Theimage-keying node 602 calls on a plurality of further processing nodesto obtain all of the input data it requires to generate the desiredimage data. In the example, the desired output image data includes aplurality of frames within which a three-dimensional computer-generatedfirst spaceship object 614 is composited within a background consistingof a clip of frames 508 portraying a ship at sea.

[0055] The image-keying node 602 therefore initially requires a viewportrendering node 604 to define a frustum and characteristics thereofwithin the three-dimensional scene, through which a two-dimensionalrendering of three-dimensional objects within a scene may take place.The image-keying node 602 subsequently requires the sequence of frames508 from frame node 605, each frame of which is processed by acolour-correction processing node 606 and a motion tracking processingnode 607, such that a composited three-dimensional first spaceshipobject 614 generated by three-dimensional modelling node 608, to whichis applied a texture 511 by the texturing node 609 and appropriatelightmaps 512 by processing node 610 and which is also motion-tracked byprocessing node 607, is seamlessly composited within the colourcorrected sequence of frames 508.

[0056] Additionally, a composited three-dimensional second spaceshipobject 617 generated by three-dimensional modelling node 618 has atexture applied to it by texturing node 616 and appropriate lightmaps bynode 615. This object is also motion-tracked by processing node 607.

[0057] In so far as the lighting of the spaceships is concerned, theimage keying processing node 602 also requires the output of a spotlightgenerated by an artificial light processing node 611 within the scene tointeract with the lightmaps 512 of first spaceship 614 and secondspaceship 617, wherein said artificial light is colour-corrected by acolour-correction processing node 612 providing a functionality similarto the functionality of colour-correction processing node 606. A filterobject is preferably generated by a light filtering processing node 613to prevent the artificial light of said spotlight from altering thecolour characteristics of the frames 508 within the scene.

[0058] In the preferred embodiment of the present invention, all of thedata generated by the above described nodes may be visualised asdistinct three-dimensional objects within a scene defined as athree-dimensional volume configured with Cartesian x, y and zco-ordinates, whereby motion tracking processing node 607 processes thex, y and z co-ordinates of each of said objects. The image-keyingprocessing node 602 subsequently overlays said three-dimensional objectsas viewed through the frustum generated by node 604.

[0059]FIG. 7

[0060] The scene data 504 generated as three-dimensional objects byscene structure 503 described in FIG. 6 are shown within a scene definedas a three-dimensional volume in FIG. 7.

[0061] The textured and lightmap-configured first spaceship model 614and second spaceship model 617 are shown within a scene 710 in relationto the viewport 711 generated by processing node 604. Said viewport isconfigured with a view frustum 701 and a focal length 702, which jointlydefine the two-dimensional plane 703 corresponding to the required imageoutput data generated by output processing node 601. Saidtwo-dimensional plane 703 may simply be thought of as the image framethat would be committed to film, were viewport 711 a conventional camerafilming three-dimensional objects within scene 710. Said two-dimensionalplane 703 will thus be hereinafter referred to as a rendering window.

[0062] The clip of frames 508 generated by node 605 is shown as atwo-dimensional plane 704 equipped with x, y and z co-ordinates withinvolume 710, wherein the area of said plane 704 is defined by theresolution in pixels or lines of the image frames 508. Said plane 704 isknown to those skilled in the art as a billboard and, depending uponwhether the functionality of window 704 allows the entire clip of frames508 to be played therein, may also be known as a player.

[0063] In the example, volume 710 also includes a spotlight object 705generated by artificial light node 611 and a filter 706 generated bynode 613. Said filter 706 is preferably positioned by motion-trackingnode 607 between spotlight 705 and player 704 in order to refractartificial light cast by said spotlight 705 within scene 710 to lightfirst spaceship 614 and second spaceship 617 and thus prevent saidartificial light from distorting the colour component values of frames508 within player 704.

[0064] The clip of image frames 508 portrays a ship 707 at sea, whereinsaid image frames were shot with a camera aboard a different ship.Consequently, the combined motions of the ship 707 and the camera aboardsaid other ship arising from random sea surface movements result in thelevel 708 of the sea to alter substantially over the entire duration ofthe clip of frames 508, whereby ship 707 in effect rises and falls alonga vertical path within player 704.

[0065] Image editor 101 wishes to remove this vertical motion in thecomposited output clip, effectively tracking a fixed portion of theframe in each frame of the clip of frames 508. However, image editor 101also wishes to composite, in each of said frames 508, first spaceship614 as moving at speed towards a position 709 and second spaceship 617moving at a slightly slower speed to position 712 while ship 707 isstationary. A problem therefore exists in that first spaceship 614requires a degree of motion blur to realistically convey the impressionof said movement at speed, second spaceship 617 requires a lesser degreeof motion blur to convey the impression of slower movement, and ship 707requires no motion blurring at all because it is stationary.

[0066] With reference to the known prior art described in FIG. 3,conventional compositing techniques would require image processingsystem 102 to generate a first clip of frames portraying first spaceship614 moving towards position 709 with motion blurring, the total numberof frames of which equals the total number of frames of the clip offrames 508. A second output clip should then be generated as portrayingsecond spaceship 617 moving towards position 712 with less motionblurring, the total number of frames of which again equals the totalnumber of frames of the clip of frames 508. Finally a third output clipshould then be generated as portraying stationary ship 707 with a stablesea level 708, for instance by processing the clip of frames 508 withthe “stabiliser with roll” disclosed in U.S. Pat. No 5,786,824 assignedto the Assignee of the present application. The required result would beachieved by keying the first, second and third output clips.

[0067] According to the current description, however, the above requiredoutput clip is generated by means of defining respective motion pathsfor each of first spaceship 614, second spaceship 617 and player 704within scene 710, respectively translating said first spaceship 614,said second spaceship 617 and player 704 along said paths over a periodof time equivalent to the duration of clip frames 508, rendering firstspaceship 614, second spaceship 617 and player 704 at intervals of saidperiod of time, which is also known to those skilled in the art assampling, and blending the resulting plurality of samples in order togenerate an output image frame or an output clip of image frames.

[0068]FIG. 8

[0069] The graphical user interface (GUI) of the application 502 isshown in FIG. 8, including a graphical representation of the scenestructure 503 and scene data 504 shown in FIGS. 5 and 6 and furtherdescribed in FIG. 7.

[0070] The GUI 801 of image processing application 502 is preferablydivided into a plurality of functional areas, most of which areuser-operable. A first area 802 displays scene structure 503 asthree-dimensional scene 710. Said scene 710 is preferably displayed withincluding scene data 504 graphically depicted as scene objects 614, 711and 701 to 708.

[0071] A cursor 803 is displayed which is user operable by means ofmouse or graphic tablet 106 and may be positioned by image editor 101over any portion of GUI 801 to select a variety of functions or taskswithin said plurality of functional areas. Thus, within scene displayarea 802, cursor 803 may be used to select a particular object, forinstance first spaceship 614, whereby a second user operable area 804conveys data properties, parameters and/or values specificallypertaining to said selected object. Preferably, second objectinformation display area 804 includes an object identifier portion 805,an object location portion 806 and an object properties portion 807.Portions 805 and 806 of second display area 804 are updated according towhich object cursor 803 selects within scene 302 and portion 807 may besubsequently interacted therewith by means of said cursor 803 to editany of the selected object's properties.

[0072] A third display area 808 comprises conventional user-operableclip navigation widgets allowing image editor 101 to respectivelyrewind, reverse play, pause, stop, play or fast forward the sequentialorder of image frames generated from scene 710 by means of renderingwindow 703. Alternatively, said navigation widgets 808 also provide thesame functionality as described above for player 704 if said player isselected as a scene object by means of cursor 803. A counter area 809 isprovided in close proximity to the clip navigation widget 808, which isdivided into hours, minutes, seconds and frames, such that theaforementioned navigation by means of navigation widgets 808 may becarried out with precision and provide a valid point of reference toimage editor 101.

[0073] A fourth display area 810 provides a conventional bar of menusoperable by means of cursor 803, which provide a variety of functionsand processes, for instance with which to load or store image data,further configure the size and contents of display areas 802, 804 and808 or, eventually, stop processing the instructions according to step409.

[0074]FIG. 9

[0075] The processing step 405 according to which scene data 504 isedited as scene objects shown in FIGS. 7 and 8 is further described inFIG. 9.

[0076] At step 901, the required output clip length is set. Said lengthmay be understood as the number of output image frames node 602 shouldrender through rendering window 703 in order to generate a completesequence of frames defining a clip. At step 902 the number of samplesper frame is set. This defines how many samples should be taken andstacked per output frame. More samples may give a better effect but takelonger to render. In this example the user sets four samples per frame.However this input and the input at step 901 may each be eitherautomatically calculated or input by user 101.

[0077] At step 903 a first object such as first spaceship 614 isselected in the scene 710, for instance by means of cursor 803 formotion data to be input. A path is subsequently defined for saidselected object within said scene 710 at step 904, for instance byselecting the “path” property of said object within object's propertiesportion 807.

[0078] In this example said path is linear and comprises a directionalvector, the origin which is defined by the x, y and z co-ordinates ofthe object selected at step 903 and the extremity of which is defined bythe x, y and z co-ordinates of said object subsequent to image editor101 dragging said object 614 to position 709, for instance with usingcursor 803 in a “click and drag” configuration, which is well known tothose skilled in the art. However, alternative paths include aspline-based curve, for instance to impart a “bobbing” motion to firstspaceship 614, or a function, for instance the tracking functiondisclosed in U.S. Pat. No. 5,786,824 referenced above, wherein saidfunction itself generates a linear or spline-based movement vector. Afurther alternative path is no movement at all, that is the x, y and zco-ordinates are equal for any time value. This might be appropriate ifship 707 were already stationary, in which case player 704 would notrequire tracking. The path configuration input at step 904 issubsequently stored at step 905 as three-dimensional locations 510specifically for the object selected at step 903. The skilled readerwill understand that there are many ways of generating or definingmovement paths, including any explicit, implicit or parametriccontinuous or non-continuous function or even a simple table of values.Any method that produces a three-dimensional position in response to aninput of a frame number would be suitable herein.

[0079] At step 906 a shutter length for the selected object is input andat step 907 the sampling rate for the object is calculated. This is theratio of the shutter length and the number of samples per frame input atstep 902, which in this example is four. For example, the user may set ashutter length of two frames for first spaceship 614 and of one framefor second spaceship 617. This would give a sampling rate of half aframe for first spaceship 614 and of a quarter of a frame for secondspaceship 617. The sampling rate defines the time value that is inputinto the motion path to create the position of each object for eachsample. The shutter length for player 704 is set to zero since the ship707 should have no motion blur.

[0080] A question is subsequently asked at step 908 as to whether motiondata, comprising a path and a shutter length, should be defined foranother object in scene 710. If the question asked at step 908 isanswered in the affirmative control is returned to step 902, wherebysaid next object may be selected and its motion data defined and stored.Alternatively, if the question of step 906 is answered in the negativethen step 405 is concluded.

[0081] According to the prior art, and with reference to FIG. 3, shutterlength and the number of samples are traditionally specified for theentire scene. Specifying the shutter length and number of samples forthe entire scene 710 according to said prior art would result inidentical motion blurring for the first and second spaceships and theship, when in fact they should have differing amounts. Therefore ashutter length is input for each object in scene 710 independently ofthe shutter length for other objects in the scene and of the totalnumber of frames required for the scene, whereby it is thus possible togenerate a two-dimensional image frame within which each of saidplurality of objects is rendered with its own degree of motion blurringsuch that, in the example, the ship 707 has no motion blurring whilstappropriate motion blurring is convincingly provided for first spaceship614 and second spaceship 617. Individual shutter lengths for each objectspecify the amount of motion blur to be applied to that object. Thelonger the shutter length, the more the blur.

[0082]FIG. 10

[0083] The processing step 903 of equipping an object in scene 710 witha motion path is graphically illustrated in FIG. 10.

[0084] With reference to the description of step 903, image editor 101first selects first spaceship object 614 with cursor 803 and drags itwithin scene 710 to location 709, whereby a linear directional vector1001 is defined with an origin, the x, y and z co-ordinates of which areobtained from the location of said object 710 before interactiontherewith, and an extremity 709, the x, y and z co-ordinates of whichwithin scene 710 are derived from the location at which the cursor 803releases first spaceship 614 after dragging. The user also specifies thenumber of frames taken for the first spaceship to reach location 709.

[0085] Preferably first spaceship 614 is selected according to step 903,whereby cursor 803 subsequently selects the path property of said objectwithin portion 807 such that application 502 initiates the vector originand continually reads the input data of mouse or graphic tablet 106,e.g. cursor 803, to define the length and orientation of said vectorwithin scene 710.

[0086] Similarly, user 101 selects the player 704 as a next object inscene 710 according to step 902, again selecting the path property ofsaid player within updated portion 807 but specifying said path as atracking function as opposed to a directional vector. Image editor 101may subsequently select a two-dimensional x, y portion 1002 of the imageframe area within player 704, whereby said portion will be tracked ineach subsequent frame in said frame display area of said player 704 bymeans of calculating a movement vector, the inverse of which will beapplied to player object 704 such that the x, y portion 1002 remainsstable in relation to the two-dimensional rendering window 703.

[0087] The user may then specify a movement path for second spaceship617 and a tracking function for player 704. Any method of inputtingpaths or retrieving saved paths is appropriate. Individual motion pathsare therefore defined for a number of objects representing scene datawithin scene 710.

[0088]FIG. 10A

[0089]FIG. 8A shows three frames of a clip of frames that could beproduced by a prior art system, or by the system described herein byspecifying no motion blurring. For the purposes of clarity, only threeoutput image frames 1011, 1012 and 1013 are shown from an output clipincluding potentially hundreds or even thousands of image frames.

[0090] For each frame the position of each object, as specified by itsmotion path, is calculated and the scene as viewed through renderingwindow 703. The time taken is counted inframes, and so for each framethe frame number is input into the motion path vector, function or tablefor each object. This gives a three-dimensional position for eachobject. As shown, spaceship 614 moves more quickly than spaceship 617.Although ship 707 is shown as stationary the position of player 704changes each frame according to the tracking function. This compensatesfor the movement of ship 707 within the player 704.

[0091] These three frames do not include motion blurring and areincluded herein to show the fast movement of first spaceship 614, theslower movement of second spaceship 617 and the lack of movement of ship707. The skilled reader will appreciate that the movement shown isexaggerated for illustration purposes, since in reality the movementfrom frame to frame, even of a very fast-moving object, is barelydetectable by the human eye.

[0092]FIG. 11

[0093] The processing steps according to which image data edited asscene objects is rendered at rendering step 406 are further detailed in

[0094]FIG. 11.

[0095] At step 1101, a frame counter is initialised with the number offrames defined by the output clip length configured at step 901 and thefirst frame of the output clip or output image frame is selected forrendering.

[0096] At step 1102 samples, the number of which is set by the input atstep 902, are rendered in order to generate the image data required tocreate the respective degree of motion blurring for each of the objectsin scene 710 to be rendered as viewed through rendering window 703. Upongenerating the required image data as samples of picture screenelements, also known as pixels, each of which having red, green and blue(RGB) colour component values, said samples are processed in order togenerate the target pixels defining the output image data, i.e. theoutput frame, at step 1103.

[0097] At step 1104 the number of the output frame generated at step1103 is subtracted from the number of frames initialised in the counterat step 1101, whereby a question is asked as to whether all the framesof the required output clip have been rendered. If the question of 1104is answered in the negative control is returned to step 1101, wherebythe next output frame is selected for rendering according to steps 1102and 1103 and the counter further decreased at step 1104, until such timeas question 1104 is answered in the affirmative whereby all of therequired output image data has been rendered.

[0098]FIG. 12

[0099] The processing steps according to which samples are processed atstep 1102 in order to generate image data to create motion blur arefurther detailed in FIG. 12.

[0100] At step 1201 the first object in the scene is selected and atstep 1202 its x, y and z co-ordinates within scene 710 are calculatedaccording to its motion path and sampling rate. For the first sample ofany frame the input into the objects' motion paths is the frame number.In this example, for the first sample of the first frame the input isone, which gives the object's starting point. However, after that theinput is given by the object's sampling rate. At step 1203 a question isasked as to whether there is another object whose position should becalculated, and if this question is answered in the affirmative thencontrol is returned to step 1201.

[0101] If it is answered in the negative then at step 1204 the sampleframe is rendered according to the view through rendering window 703 andat step 1205 the sample is stacked in memory. At step 1206 a question isasked as to whether there is another sample to be rendered and if thisquestion is answered in the affirmative then control is returned to step1201 and the first object is selected to have its position recalculated.In this example, first spaceship 614 has a sampling rate of half a frameand so its position half a frame after its starting position iscalculated. Second spaceship 617 has a sampling rate of a quarter of aframe and so its position a quarter of a frame after its startingposition is calculated. Player 704 has a sampling rate of zero and soits position does not move.

[0102] If the path of an object is a function then usually it willaccept an input of a fraction. However some paths, for example lookuptables, may need interpolation to provide the sub-frame positions.Additionally, for the final frame extrapolation beyond the end of thepath may be necessary.

[0103] If the question asked at step 1206 is answered in the negative,to the effect that the total number of samples for the frame has beentaken, then at step 1207 the samples are superimposed upon one anotherto produce the output frame.

[0104]FIG. 13

[0105] A graphical representation of the sampling and stacking ofprocessing steps 1204, 1205 as well as the generation of the outputframe at step 1207 is provided in FIG. 13.

[0106] In the example, it was previously explained that ship 707 shouldhave no motion blurring whatsoever whereas first spaceship 614 andsecond spaceship 617 require different degrees of motion blurring toconvey the impression of movement at speed. Moreover, in the preferredembodiment of the present invention, only a tracked portion of player704 is required for the final output image data.

[0107] Consequently, image editor 101 specifies a shutter length of zeroframes for player object 704. This gives a sampling rate of zero, sothat the x, y and z position of said player object 704 will only becalculated for whole frame numbers and thus during a single frame theobject will appear immobile regardless of the number of samples takenand stacked according to steps 1204 and 1205. Using the analogy of acamera, it is of course meaningless to specify that a shutter is openfor no time and yet generates an image. However, in the current context,if the number of samples taken is over the smallest fraction of a framepossible then all the samples will be virtually identical. Thus theshutter length for object 704 can be thought of as tending to zero. Whenthe next frame is sampled the player's position will be calculated byinputting that frame number into the tracking function that keeps ship707 stationary and it will be in this position for all the samples takenfor the next frame.

[0108] Conversely, image editor 101 specifies a shutter length of twoframes for first spaceship object 614, such that the number of samplesspecified for the scene at step 907 is equally divided between twoframes' worth of the motion path. Also, user 101 specifies a shutterlength of one frame for second spaceship 617, which means that thenumber of samples is only taken from a single frame's worth of the path.

[0109] This system gives uniform sampling. Weighted sampling is achievedby using a more complex calculation of the sampling rate than a simpleratio, usually requiring a weighting constant as well as the shutterlength and number of samples per frame. For example, more samples couldbe taken near to the end of the shutter length than at the beginning.The concepts of both uniform sampling and weighted sampling will befamiliar to those skilled in the art and, for the purpose of clarity,only uniform sampling will be described in the present description.Alternatively, using uniform sampling but a weighted average for thegeneration of the output frame gives a similar effect to weightedsampling.

[0110] In the example, at step 902 it is specified that the number ofsamples per frame to be taken and stacked to generate each output imageframe is four. Thus four samples are taken and stacked according tosteps 1204 and 1205. These are shown at 1301, 1302, 1303 and 1304. Ship707 is stationary for all four samples since it has a shutter length ofzero.

[0111] In sample 1301 both spaceships are at the origin of their paths,since the input into their motion path functions is one. In sample 1302spaceship 614 is at a position halfway between its position in the firstframe of the clip, as shown in FIG. 10A at 1011, and the second frame,as shown at 1012. In sample 1303 it is at the same position as in frame1012 and in sample 1304 it is at a position halfway between frame 1012and the third frame, frame 1013. Thus four samples are taken ofspaceship 614 over a shutter length of two frames, that is at one frame,one and a half frames, two frames and two and a half frames.

[0112] Spaceship 617 has a shutter length of one frame and so in sample1302 it is at a position of a quarter of a frame after its startingpoint, in sample 1303 it is at a position of half a frame after itsstarting point and in sample 1304 it is at a position of three quartersof a frame after its starting point.

[0113] Thus frame 1305 is the superposition of the four samples 1301 to1304, the sum total of the colour component values of each correspondingpixel of which is divided by the total number of samples to provide asingle image frame comprising the target pixels. Ship 704 is stationary,second spaceship 617 has a small amount of motion blur and firstspaceship 614 has a lot of motion blur.

[0114] In this example the interpolated samples were taken after thebeginning of each frame. In another embodiment of the invention it ispossible for the user to specify a value for phase, to indicate whetherthe samples are taken before, around or after the frame. In this caseextrapolation could be needed to obtain positions before the firstframe.

[0115] Thus there is provided apparatus for generating image data,comprising memory means configured to store data defining a volumecomprising a plurality of objects and at least one viewport, memorymeans configured to store motion data for each of said objects,configuration data for said viewport and instructions, and processingmeans configured by said instructions to perform the following steps.For each of the objects a position is calculated along its motion pathat an interval of a user-specified time period, wherein the interval isdependent upon the shutter length for the object. The object is thentranslated to this position. The objects are then rendered through theviewport to produce a sample. When a specified plurality of samples havebeen rendered they are blended to generate image data wherein eachobject is independently motion blurred.

[0116]FIG. 14

[0117] The processing steps according to which application 502 processesthe samples stacked according to step 1205 to generate output frameimage data 1310 at step 1206 are further detailed in FIG. 14.

[0118] At step 1401, the first target pixel of the output image frame1310 is selected, the respective red, green and blue colour componentvalues of which should equal the average of the respective red, greenand blue colour component values of the pixels having the sametwo-dimensional co-ordinates in all of the samples stacked.

[0119] In order to generate said average, each sample is recursivelyselected at step 1402 such that the pixel therein having x, yco-ordinates equivalent to the x, y co-ordinates of the selected targetpixel of step 1401 may be selected at step 1403 and its respectivecolour component values added at step 1404. At step 1405 a firstquestion is asked as to whether all of the samples have been traversedand processed, thus providing total colour component values. If thisquestion is answered in the affirmative then at step 1406 these totalsare divided by the total number of samples traversed to obtain the finalRGB colour component value of the target pixels selected at step 1401.Alternatively, if the question of step 1405 is answered in the negative,the next sample is selected at step 1402, traversed to identify thecorresponding pixel at step 1403, the RGB values of which are addedaccording to step 1404.

[0120] Having obtained final RGB colour component values for the targetpixel currently selected at step 1401, a second question is asked atstep 1407 as to whether all of the target pixels defining the targetimage frame have been processed, such that the next output image framemay be generated for the final output clip. If the question of step 1407is answered in the negative control returns to step 1401, whereby thenext pixel of the target image frame is selected and its final RGBcolour component values calculated according to step 1402 to 1406.

[0121] Question 1407 is eventually answered in the affirmative, wherebyan output image frame is generated which includes a plurality ofobjects, each of which has its own degree of motion blurring and istherefore independently motion blurred. In this example the degree ofmotion blurring of ship 707 is nil, but it can still considered to be anobject to which motion blurring is applied, simply with a shutter lengthof zero.

[0122] If spaceships 614 and 617 had the same shutter length specifiedfor them, for example one frame, then they would have the same degree ofmotion blurring. However, they would still be independently motionblurred since their sampling rates and positions would have beencalculated independently of each other.

[0123]FIG. 15

[0124] Three frames of a clip of motion-blurred frames produced byrendering the objects in scene 710 at step 406 are shown in FIG. 15.These are the same frames as shown in FIG. 10A but with motion blurringapplied.

[0125] The first frame 1305 is generated as shown in FIG. 13. The secondframe 1501 is generated by stacking samples with the first spaceship atpositions of two frames, two and a half frames, three frames and threeand a half frames and the second spaceship at positions of two frames,two and a quarter frames, two and a half frames and two and threequarters frames. Player 704 is at a position of two frames throughout.The third frame 1502 is generated similarly.

[0126] Thus it can be seen that the moti on blurring of spaceship 614overlaps from frame to frame, whereas the motion blurring of spaceship617 does not because it has a shutter speed of only one frame.

1. Apparatus for generating image data, comprising: memory meansconfigured to store data defining a volume comprising a plurality ofobjects and at least one viewport, memory means configured to storemotion data for each of said objects, configuration data for saidviewport and instructions, and processing means configured by saidinstructions to perform the steps of: (a) for each of said objects,defining said motion data in response to user input as a shutter lengthand a motion path within said volume; (b) for each of said objects,calculating a position along said motion path at an interval of time,wherein said interval is dependent upon said shutter length for saidobject, and translating said object to said position within said volume;(c) rendering said objects through said viewport to produce a sample;(d) repeating steps (b) and (c) a user-defined number of times atdifferent intervals to produce a number of samples; and (e) blendingsaid samples to generate an image frame wherein each of said objects isindependently motion-blurred.
 2. Apparatus according to claim 1, whereinsaid motion path is defined by a continuous function.
 3. Apparatusaccording to claim 1, wherein said motion path is defined by a lookuptable.
 4. Apparatus according to claim 1, wherein said configurationdata for said viewport comprises a view frustum and a focal length. 5.Apparatus according to claim 1, wherein for each of said objects saidinterval of time is a fraction of a frame.
 6. Apparatus according toclaim 5, wherein for each of said objects said fraction of a frame iscalculated by dividing said shutter length by said user-defined number.7. A method for generating image frames from data defining a volumeincluding a plurality of objects and at least one viewport, comprisingthe steps of (a) for each of said objects, defining said motion data inresponse to user input as a shutter length and a motion path within saidvolume; (b) for each of said objects, calculating a position along saidmotion path at an interval of time, wherein said interval is dependentupon said shutter length for said object, and translating said object tosaid position within said volume; (c) rendering said objects throughsaid viewport to produce a sample; (d) repeating steps (b) and (c) auser-defined number of times at different intervals to produce a numberof samples; and (e) blending said samples to generate an image framewherein each of said objects is independently motion-blurred.
 8. Amethod according to claim 7, wherein said motion path is defined by acontinuous function.
 9. A method according to claim 7, wherein saidmotion path is defined by a lookup table.
 10. A method according toclaim 7, wherein said configuration data for said viewport comprises aview frustum and a focal length.
 11. A method according to claim 7,wherein for each of said objects said interval of time is a fraction ofa frame.
 12. A method according to claim 11, wherein for each of saidobjects said fraction of a frame is calculated by dividing said shutterlength by said user-defined number.
 13. A computer readable mediumhaving computer readable instructions executable by a computer, suchthat said computer performs the steps of: (a) defining image data as avolume including a plurality of objects and at least one viewport; (b)for each of said objects, defining a shutter length and a motion pathwithin said volume in response to user input; (c) for each of saidobjects, calculating a position along said motion path at an interval oftime, wherein said interval is dependent upon said shutter length forsaid object, and translating said object to said position within saidvolume; (d) rendering said objects through said viewport to produce asample; (e) repeating steps (c) and (d) a user-defined number of timesat different intervals to produce a number of samples; and (f) blendingsaid samples to generate an image frame wherein each of said objects isindependently motion-blurred, defining a motion path for each of saidobjects within said volume in reply to user input; translating saidobject along said path over a user-specified period of time; renderingsaid object through said viewport at portions of said user-specifiedtime period; and blending the resulting plurality of rendered objects togenerate said image data.
 14. A computer-readable medium according toclaim 13, wherein said motion path is defined by a continuous function.15. A computer-readable medium according to claim 13, wherein saidmotion path is defined by a lookup table.
 16. A computer-readable mediumaccording to claim 13, wherein said configuration data for said viewportcomprises a view frustum and a focal length.
 17. A computer-readablemedium according to claim 13, wherein for each of said objects saidinterval of time is a fraction of a frame.
 18. A computer-readablemedium according to claim 17, wherein for each of said objects saidfraction of a frame is calculated by dividing said shutter length bysaid user-defined number.
 19. A computer system programmed to generateimage data, including memory means configured to store data as a volumecomprising at least one object and at least one viewport, memory meansconfigured to store motion data for said object, configuration data forsaid viewport and instructions, and processing means configured by saidinstructions to perform the steps of (a) for each of said objects,defining said motion data in response to user input as a shutter lengthand a motion path within said volume; (b) for each of said objects,calculating a position along said motion path at an interval of time,wherein said interval is dependent upon said shutter length for saidobject, and translating said object to said position within said volume;(c) rendering said objects through said viewport to produce a sample;(d) repeating steps (b) and (c) a user-defined number of times atdifferent intervals to produce a number of samples; and (e) blendingsaid samples to generate an image frame wherein each of said objects isindependently motion-blurred.
 20. A computer system programmed accordingto claim 19, wherein said motion path is defined by a continuousfunction.
 21. A computer system programmed according to claim 19,wherein said motion path is defined by a lookup table.
 22. A computersystem programmed according to claim 19, wherein said configuration datafor said viewport comprises a view frustum and a focal length.
 23. Acomputer system programmed according to claim 19, wherein for each ofsaid objects said interval of time is a fraction of a frame.
 24. Acomputer system programmed according to claim 23, wherein for each ofsaid objects said fraction of a frame is calculated by dividing saidshutter length by said user-defined number.